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ABSTRACT 


Software  project  development  has  been  plagued  with  an  infamous 
reputation  for  cost  overruns,  late  deliveries,  poor  reliability  and 
users'  dissatisfaction.  Much  of  this  blame  has  been  placed  on  the 
manner  in  which  software  development  projects  are  managed.  The 
System  Dynamics  Model  of  Software  Project  Management  is  a 
quantitative  model  of  software  project  dynamics  that  is  attempting 
to  gain  some  valuable  insight  into  the  managerial  side  of 
developing  software  systems. 

The  objective  of  this  thesis  is  to  use  the  System  Dynamics 
Model's  gaming  interface  to  investigate  the  effects  of  feedback  on 
software  project  managers.  Specifically,  subjects  were  provided 
with  either  feedforward,  outcome  feedback,  or  cognitive  feedback  to 
determine  which  feedback  form,  if  any,  improved  the  subjects' 
performance  when  confronted  with  a  complex  dynamic  task,  such  as 
software  project  management.  The  results  show  that  subjects  in  the 
cognitive  feedback  condition  achieve  a  higher  level  of  performance 
than  those  in  either  the  feedforward  or  outcome  feedback 
conditions . 
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INTRODUCTION 


I  . 


A .  BACKGROUND 

The  proliferation  of  computing  equipment  over  the  past 
years  has  served  to  increase  the  demand  for  more  reliable  and 
complex  software.  Unfortunately,  the  success  that  has  been 
common  to  the  hardware  industry  has  not  been  shared  by  those 
in  the  i-oftware  industry.  Today's  software  projects  are 
typically  delivered  late  and  over  budget.  These  inaccuracies 
have  been  blamed,  in  part,  on  ineffective  software  project 
managers.  (Schlender,  1989) 

A  large  portion  of  these  inaccuracies  associated  with  the 
general  project  management  problem  can  be  attributed  to  the 
difficulty  of  control.  One  basic  element,  evident  in  any 
control  system,  is  a  means  of  transmitting  feedback 
information  to  the  control  device  (Anthony  and  Dearden,  1980, 
pp.  3-4).  Control  relies  heavily  on  information  feedback;  the 
question  is,  however,  what  kind  of  feedback? 

There  has  been  a  great  deal  of  research  analyzing  the 
effects  of  outcome  feedback  on  management  control,  but  this 
type  of  feedback  has  not  been  effective  in  improving  the 
performance  of  decision  makers.  Research  in  static  situations 
shows  cognitive  feedback  to  be  more  effective  than  outcome 
feedback  in  enhancing  decision  quality.  However,  little  work 
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has  been  done  to  determine  how  cognitive  feedback  may  assist 
management  control  in  complex  dynamic  tasks. 

Researchers  have  also  suggested  that  the  performance  of  a 
decision  maker  in  a  dynamic  decision  task,  such  as  software 
project  management,  would  improve  if  the  decision  maker's 
model  matches  that  of  the  task.  Therefore,  providing 
individuals  with  feedforward  on  a  task  may  improve  decision 
quality  as  opposed  to  outcome  feedback.  The  focus  of  this 
thesis  is  on  studying  the  effects  of  outcome  feedback, 
cognitive  feedback,  and  feedforward  on  one  particular 
management  control  problem,  that  of  software  project 
management . 


B.  EXPERIMENTATION  TOOL 

The  System  Dynamics  Model  of  Software  Project  Management 
(SDM)  is  a  comprehensive  model  of  the  software  development 
process  that  integrates  both  the  managerial  and  software 
development  activities.  Through  the  use  of  a  model. 

The  effects  of  different  assumptions  and  environmental 
factors  can  be  tested.  In  the  model  system,  unlike  the 
real  systems,  the  effect  of  changing  one  factor  can  be 
observed  while  all  other  factors  are  held  unchanged.  Such 
experimentation  will  yield  new  insights  into  the 
characteristics  of  the  system  that  the  model  represents. 

By  using  a  model  of  a  complex  system,  more  can  be  learned 
about  internal  interactions  than  would  ever  be  possible 
through  manipulation  of  the  real  system.  Internally,  the 
model  provides  complete  control  of  the  system's 
organizational  structure,  its  policies,  and  its 
sensitivities  to  various  events.  (Forrester,  1961,  p.l) 
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Additionally,  this  particular  model  provides  an  effective 
means  of  studying  dynamic  decisions. 

The  gaming  interface  of  the  System  Dynamics  Model  provides 
experimenters  with  the  ability  to  analyze  the  efforts  of  any 
number  of  software  project  managers.  The  experimenter  can 
vary  the  type  of  feedback  given  to  the  manager  by  specifically 
tailoring  the  model's  interface  for  that  particular  manager. 
The  model  provides  the  capability  of  displaying  a  wide  variety 
of  variables  in  either  tabular  ot  graphical  form.  The  results 
of  each  manager's  run  can  then  be  collected  and  analyzed  to 
determine  any  particular  trends  in  their  decision  making 
process . 

C .  RESEARCH  QUESTION 

Recent  laboratory  experiments  have  provided  valuable 

insight  into  human  behavior  in  a  variety  of  decision- theoretic 

contexts.  This  research,  however,  has  focused  mainly  on 

static  and  discrete  judgements.  As  Hogarth  (1981)  emphasizes 

...the  continuous,  adaptive  nature  of  the  judgmental 
processes  used  to  cope  with  a  complex,  changing 
environment....  With  few  exceptions ...  judgment 

researchers  have  focused  on  discrete  incidents  (particular 
actions,  predictions,  and  choices)  that  punctuate  these 
continuous  processes;  furthermore,  task  environments  are 
typically  conceptualized  to  be  stable,  (p.  198) 

Sterman  (1989a)  argues  that  experimental  studies  of  the 
"continuous,  adaptive  nature  of  judgmental  processes"  in  a 
dynamic  system,  such  as  software  project  management,  can  be 
conducted  in  the  laboratory  with  the  aid  of  computer 
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simulation  models.  He  adds  that  "simulations  can  represent 
the  structure  and  complexity  of  such  systems  with  great 
fidelity  and  permit  controlled  manipulations  of  the  decision 
context  and  information  presented  to  the  subject." 

As  an  example,  the  research  question  addressed  in  this 
thesis  is:  What  effects  do  cognitive  feedback,  outcome 
feedback,  and  feedforward  have  on  decision  makers  in  a  dynamic 
decision  environment  such  as  software  project  management? 

D.  CONTRIBUTION 

Enhancement  of  management  control  through  the  use  of 
cognitive  feedback  has  attracted  much  attention  (Kleinmuntz, 
1985;  Sterman,  1989a).  The  use  of  cognitive  feedback  to  aid 
software  project  managers  has  not,  however,  been  investigated. 
The  goal  of  this  research,  therefore,  is  to  establish  the 
importance  of  cognitive  feed-back  as  an  aid  to  the  decision 
making  process  of  software  project  managers. 
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II.  THEORETICAL  PREMISE 


A.  FEEDBACK  IN  A  DYNAMIC  DECISION  ENVIRONMENT 

1.  Static  vs  Dynamic  Decision  Environments 

When  analyzing  human  judgmental  ability,  it  is 

important  to  distinguish  between  static  and  dynamic  decision 

environments.  Although  much  of  human  decision  making  is 

composed  of  discrete  incidents  (particular  actions, 

predictions,  and  choices)  occurring  in  a  seemingly  static 

environment,  these  incidents  are  only  a  subset  of,  and  serve 

to  punctuate,  our  continuous  processes  which  occur  in  response 

to  our  dynamic  environment.  As  Hogarth  (1981)  indicates,  the 

limitation  of  existing  research  on  human  judgment  is  that  it 

focuses  only  on  these  discrete  incidents  in  static 

environments.  Since  most  decisions  are  made  in  a  continuous, 

dynamic  environment,  it  is  argued  that  biases  observed  during 

these  discrete  incidents  occur  as  a  result  of  heuristics  that 

are  derived  from  man's  more  natural  continuous  environment. 

According  to  Hogarth  (1981),  failure  to  evaluate  human 

judgement  as  a  continuous  process  has  two  distinct  pitfalls: 

First,  insufficient  attention  has  been  paid  to  the  effects 
of  feedback  between  organism  and  environment.  Second, 
although  judgmental  performance  has  been  evaluated 
according  to  the  principles  of  optimal  behavior  implied  by 
decision  theory  and  the  probability  calculus,  few 
researchers  have  questioned  whether  the  assumptions  of 
such  models  apply  to  continuous  processes,  (p.  198) 
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Studies  involving  human  decision  making  cannot 
overlook  the  importance  of  feedback.  Most  all  human  judgement 
is  used  to  facilitate  an  action  and  this  action  is  most  often 
followed  by  immediate  feedback.  Our  next  action  is  then 
directly  influenced  by  this  feedback  causing  a  action, 
outcome,  feedback,  action  loop.  Hogarth  (1981)  indicates  that 
the  tendency  to  overlook  feedback  as  a  crucial  part  of  this 
loop  comes  as  a  result  of  its  "ubiquity"  in  the  environment. 
As  Powers  states  (1973); 

All  behavior  involves  strong  feedback  effects,  whether  one 
is  considering  spinal  reflexes  or  self-actualization. 
Feedback  is  such  an  all-pervasive  and  fundamental  aspect 
of  behavior  that  it  is  as  invisible  as  the  air  we  breathe. 
Quite  literally  it  is  behavior--we  know  nothing  of  our  own 
behavior  but  the  feedback  effects  of  our  own  outputs,  (p. 
351) 

2.  Inadequacy  of  Outcome  Feedback 

A  majority  of  the  work  that  has  been  done  in  relation 
to  the  dynamic  decision  environment  has  examined  the  effects 
of  outcome  feedback  on  the  decision  making  process  (Brehmer, 
1987;  Sterman,  1989b).  Evidence,  however,  indicates  that 
presenting  outcome  feedback  in  a  dynamic  environment  has 
dysfunctional  effects  that  persist  over  time.  These  effects 
fall  into  four  categories. 

First,  subjects  typically  misperceive  time  lags  in  the 
system  which  confronts  them.  In  Sterman 's  (1989b)  stock 
management  problem,  subjects  fail  to  adequately  account  for 
the  supply  line.  Subjects  confronted  with  Brehmer  s  (1987) 
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DESSY  experiment  show  improvement  after  spending  two  hours  a 
day  for  four  days  with  the  system,  "but  only  if  there  are  no 
delays  in  the  system.  If  there  are  even  minimal  delays,  the 
subjects'  control  over  the  system  does  not  improve."  As 
Brehmer  states,  "this  [fact!  is  somewhat  disconcerting,  since 
delays  are  probably  a  more  common  case  than  that  of  immediate 
feedback. " 

The  second  dysfunctional  effect  that  typically  plagues 
subjects  in  outcome  feedback  experiments  is  a  wide  oscillation 
of  results  over  time  (Sterman,  1989b).  In  Sterman's  stock 
management  problem,  this  oscillation  is  seen  in  the  inventory 
level.  Subjects  in  Sterman's  experiment  also  attribute  the 
dynamics  of  the  system  to  external  variables  rather  than  as  a 
direct  result  of  their  interactions  with  the  environment. 
Thus,  subjects  misperceive  the  feedback  from  their  own 
decisions.  The  final  dysfunctional  effect  of  outcome  feedback 
is  seen  in  (Wagenaar,  1985)  where  subjects  misperceived 
exponential  growth  over  time  (and  hence,  nonlinear  changes). 

Experimental  evidence  indicates  that  outcome  feedback 
is  not  an  adequate  aid  in  decision  making.  As  Sterman  (1989b) 
states:  "The  results  here  suggest  that  outcome  feedback  alone 
is  not  sufficient:  by  attributing  the  source  of  change  to 
external  factors,  people's  mental  models  lead  them  away  from 
the  true  source  of  difficulty."  Kleinmuntz  and  Thomas  (1987) 
drew  similar  conclusions:  "Despite  the  corrective  benefits  of 
outcome  feedback. . . ,  it  may  still  be  quite  difficult  to  learn 
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how  to  improve  one's  decision  rules  using  outcome  feedback 
alone."  The  inadequacy  of  outcome  feedback  in  dynamic 
environments  has  led  researchers  to  explore  alternative  means 
of  improving  performance. 


B.  ALTERNATIVES  TO  OUTCOME  FEEDBACK 

1.  Cognitive  Feedback 

In  contrast  to  outcome  feedback,  which  provide 
subjects  with  information  about  the  accuracy  or  correctness  of 
their  response,  cognitive  feedback  represents  "information 
regarding  the  how  or  why  that  underlies  this  accuracy." 
(Jacoby  et  al.,  1984)  Doherty  and  Balzer  (1988)  describe 
cognitive  feedback  as  information  which  provides  subjects  with 
the  following  relationships: 


1.  Between  cues  and  criterion,  i.e.,  information  about  the 

task.  This  is  known  as  task  information  and  it  is 

characterized  by  three  kinds  of  relational  indices-- 
overall  task  predictability  (Re),  cue  intercorrelations 
(rij),  and  correlations  between  cues  and  the  criterion 
(rie) . 

2.  Between  cues  and  the  person's  inferences,  i.e. 
information  about  the  person's  cognitive  state.  This  is 
known  as  cognitive  information  and,  in  terms  of  the  lens 
model,  largely  mirrors  task  information  (the  only 
exception  being  cue  intercorrelations  which  is  not 
represented  in  this  relationship). 

3.  Between  cognitions  and  the  distal  objects.  This  third 
category  comprises  indices  of  "functional  validity" 
information,  or,  information  about  the  relation  of  the 
cognitive  system  to  the  task  system  (Balzer  et  al., 
1989).  These  indices  include  the  achievement  correlation 
(ra),  the  matching  index  (G),  and  the  correlation  between 
the  residuals  from  the  predictions  of  those  models  (C). 
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Since  Cognitive  Feedback  has  been  used  largely  for 
static  tasks  in  the  context  of  the  lens  model,  the  aim  of  this 
study  is  to  extend  this  notion  to  a  dynamic  decision  situation 
using  a  system  dynamics  model.  As  Sterman's  (1989b)  research 
indicates  "the  efficacy  and  robustness  of  decision  strategies 
lies  not  only  in  the  availability  of  outcome  feedback,  but 
depends  crucially  on  the  nature  of  the  action  feedback  between 
decisions  and  changes  in  the  environment  which  condition 
future  decisions."  Brehmer  (1987)  concludes  from  his  DESSY 
experiment  that,  "results  on  verbalization  suggest  that 
information  about  the  system  may  need  to  be  communicated  in 
nonverbal  form,  and  that  various  graphic  displays  may  prove 
useful."  Additionally,  Kleinmuntz  and  Thomas  (1987)  state 
that  "feedback  about  the  decision  process  being  used  may  also 
be  valuable...."  Each  of  these  statements  support  the  belief 
that  cognitive  feedback  should  prove  more  beneficial  than 
outcome  feedback  when  presented  to  subjects  confronted  with  a 
dynamic  decision  situation. 

2 .  Feedforward 

Another  method  of  assisting  subjects  in  forming  mental 
models  of  complex  systems  is  through  feedforward.  Feedforward 
can  be  defined  as  "the  transmission  of  task  information 
directly  to  the  subject."  (Bjorkman,  1972)  Studies  by  Conant 
and  Ashby  (1970)  showed  that  in  order  to  perform  effectively 
with  a  dynamic  process,  an  operator  must  have  a  model  of  the 
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system.  Bjorkman  (1972)  provides  three  hypotheses  with  regard 
to  the  use  of  feedforward  in  knowledge  and  policy  formation: 


1.  Knowledge  acquired  by  feedforward  should  be  more  accurate 
and  consistent  since  it  does  not  suffer  from  various 
sources  of  error  and  bias  due  to  the  trial--by--trial 
accumulation  of  information. 

2.  Feedforward  relieves  the  learner  from  a  certain  amount  of 
cognitive  strain  since  he  already  knows  things  which 
otherwise  should  have  been  learned  by  feedback.  This  may 
give  the  learner  an  increased  opportunity  to  focus  on 
policy  formation. 

3.  It  seems  reasonable  to  assume  that  feedfor  rd  favors  an 
analytical  rather  than  intuitive  mode  of  thought. 
Uncertainty,  which  is  one  of  the  factors  that  contributes 
to  intuitive  rather  than  logical,  stepwise  inference,  has 
been  removed  entirely  or  partly  by  feedforward. 


The  operationalization  of  feedforward  occurred,  for  the 
purposes  of  this  study,  through  the  use  of  a  special  training 
session  which  assisted  subjects  in  forming  a  model  of  the 
software  project  management  system. 

3 .  Cognitive  Feedback  vs  Feedforward 

Although  feedforward  assists  subjects  in  forming  a 
mental  model  of  the  system,  task  information  is  presented  only 
prior  to  performing  the  task  (or,  at  best,  the  information  is 
presented  prior  to  performing  the  task  and  the  same 
information  is  available  to  the  subject  throughout  the  task) . 
As  shown  by  Morris  and  Rouse  (1986),  "a  priori  knowledge  can 
be  a  powerful  basis  for  gaining  new  knowledge  or,  if 
incorrect,  an  impediment  to  gaining  correct  knowledge."  A 
distinct  advantage  gained  through  cognitive  feedback  is  that 
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subjects  are  constantly  being  presented  with  task  information 
so  that  they  may  change  their  mental  models  of  the  system  to 
meet  changes  experienced  in  the  environment. 

C .  HYPOTHESES 

Two  primary  hypotheses  guide  the  research  question.  The 
first  hypothesis  is  concerned  with  the  relationship  between 
feedback  condition  and  subject  performance.  As  indicated  in 
sections  1  and  2,  cognitive  feedback  and  feedforward  assist 
subjects  in  forming  a  mental  model  of  the  task.  Thus, 
subjects  receiving  this  information  should  perform  better  than 
those  not  receiving  this  information.  Additionally,  cognitive 
feedback  continually  assists  subjects  in  keeping  their  mental 
model  current  with  the  dynamic  environment.  One  would 
therefore  expect  subjects  receiving  cognitive  feedback  to  also 
outperform  those  subjects  receiving  feedforward.  This  leads 
us  to  the  first  hypothesis: 

Subjects  receiving  cognitive  feedback  will  perform  better 
than  subjects  receiving  either  outcome  feedback  or 
feedforward. 

The  second  primary  hypothesis  addresses  the  characteristic 
of  the  task  which  confronts  the  subject.  This  is  accomplished 
by  measuring  each  subjects  performance  during  each  of  three 
software  projects  with  varying  degrees  of  complexity  (details 
on  the  three  projects,  referred  to  as  ideal,  f ixedsize/bad 
estimates,  and  undersize  will  be  provided  in  Chapter  III).  We 
would  expect  that  subjects  would  perform  better  when 
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confronted  with  a  task  with  less  complexity.  Since  the  ideal 

project  has  fewer  lags  and  less  of  the  "noise"  associated  with 

more  complex  tasks,  it  is  regarded  as  the  least  complex  of  the 

three  projects.  The  second  hypothesis  is  therefore: 

Subjects  performance  while  managing  the  ideal  software 
project  will  be  better  than  when  managing  either  the 
f ixedsize/bad  estimates  or  undersize  software  projects. 
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Ill . 


METHOD 


A.  TASK  ENVIRONMENT 

The  task  that  subjects  were  asked  to  perform  was  in  many 
ways  similar  to  flight  simulators  that  pilots  use  to  mimic 
flying  an  aircraft  from  takeoff  at  point  A  to  landing  at  point 

B.  Instead  of  flying  an  aircraft,  though,  the  simulation 
mimicked  the  life  of  three  real  software  projects  from  the 
start  of  the  design  phase  until  the  end  of  testing.  Subjects 
were  more  than  outside  observers,  however,  they  performed  an 
actual  role  in  the  project;  that  of  the  project  manager. 

Specifically,  subjects  were  required  to  track  each 
project's  progress  using  a  number  of  reports  generated  by  the 
project  team  at  different  intervals  throughout  the  project 
life.  They  then  made  project  staffing  decisions  based  on  the 
knowledge  gained  from  those  reports.  As  project  manager, 
subjects  were  permitted  to  hire  additional  staff  or  decrease 
the  staffing  level  as  deemed  necessary  to  complete  the 
project.  Their  objective  in  setting  the  staffing  level  was  to 
decide  on  the  best  compromise  between  finishing  on  an 
acceptzdsle  schedule  while  avoiding  an  excessive  cost  overrun. 
Specifically,  subjects  attempted  to; 

1.  complete  the  project  on  schedule, 

2.  at  the  lowest  possible  cost,  and 
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3.  in  any  case,  complete  it  before  the  maximum  tolerable 
completion  date. 

Unlike  Sterman's  (1989b)  use  of  a  Generic  Stock-Management 
System  and  Brehmer's  (1987)  DESSY  experiments  which  presented 
subjects  with  open-ended  tasks,  the  task  which  confronted  each 
subject  in  our  experiment  was  close-ended  with  each  project 
having  a  finite  completion  point. 

The  task  of  managing  a  Software  Project  was  selected  for 
several  reasons.  First,  Software  Project  Management  has  all 
of  the  characteristics  of  a  dynamic  problem  (Brehmer  and 
Allard,  1985 ) ; 


1.  It  requires  a  series  of  decisions. 

2.  The  environment  changes  both  spontaneously  (staff 
productivity,  changing  requirements,  etc)  and,  as  a 
consequence  of  the  decision  maker's  actions. 

3.  The  time  element  is  critical;  it  is  not  enough  to  make 
the  correct  decisions  and  to  make  them  in  the  correct 
order,  they  also  have  to  be  made  at  the  correct  moment  in 
time . 


Like  Brehmer's  (1987)  DESSY  experiment,  the  Software  Project 

Management  problem  is  interesting 

. . .because  the  standard  normative  theories  for  decision 
making  do  not  apply  (Brehmer  and  Allard,  1985);  the  models 
of  the  task  embodied  in  these  theories  simply  do  not  fit 
this  kind  of  task.  It  is  not  possible  to  compute  the 
correct  course  of  action.  This  can  only  be  found  from  a 
model  of  the  system  and,  before  the  operators  have 
developed  such  models,  they  will  not  be  able  to  control 
the  system.  The  research  problem,  is  whether  or  not 
people  are  able  to  develop  good  mental  models  of  this  and 
similar  tasks,  (p.  24) 
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Finally,  Software  Project  Management  is  currently  a  critical 
issue  due  to  the  frequency  of  projects  that  are  delivered  over 
budget  and  late  (Schlender,  1989). 

Three  separate  projects,  referred  to  as  ideal, 
f ixedsize/bad  estimates,  and  undersize,  were  selected  in  an 
attempt  to  cover  the  spectrum  of  projects  that  typically 
confront  Software  Project  Managers.  Table  1  shows  the 
characteristics  associated  with  each  of  the  three  projects. 


TABLE  1.  PROJECT  CHARACTERISTICS 


Ideal 

Undersize 

Fixedsize 

Project  cost,  initial  estimate 
(Man-days ) 

3,721 

1 ,460 

2,972 

Project  Size,  initial  estimate 
( No .  of  tasks  I 

1,067 

397 

1,866 

Actual  Size  of  Project 
( No .  of  tasks  1 

1,067 

610 

1,866 

Project  duration!  initial  estimate 
( Days ) 

413 

362 

380 

Maximum  tolerable  project  duration 
( Days  1 

479 

420 

441 

Noteai  1.  The  ideal  project  had  accurate  initial  estimates  of  project  size  and  cost. 

2.  The  undersize  project  had  understated  initial  estimates  of  size,  and  therefore, 
cost. 

5.  The  fixedsize  project  had  an  accurate  initial  estimate  of  size.  The  initial  cost 
estimate  uas,  houever,  understated. 


Subjects  were  presented  with  accurate  initial  estimates  as 
well  as  accurate  information  throughout  the  entire  lifecycle 
of  the  ideal  project.  Subjects  were  given  accurate  initial 
estimates  for  the  f ixedsize/bad  estimate  project,  however, 
estimates  given  during  the  project  lifecycle  were  typically 
unreliable.  The  undersize  project  was  a  project  that  grew  in 
size  from  an  initial  estimate  of  397  tasks  to  610  tasks  at 
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project  completion.  This  growth  in  project  size  was 
attributable  to  changing  user  requirements. 

B .  MODEL 

The  Model  of  Software  Project  Management  attempts  to 
provide  "a  comprehensive  model  of  the  dynamics  of  software 
development  that  enhances  our  understanding  of,  provides 
insight  into,  and  makes  predictions  about  the  process  by  which 
software  development  is  managed."  (Abdel-Hamid  and  Madnick, 
1989)  Figure  1  shows  the  model  with  its  four  subsystems:  the 
human  resource  management  subsystem,  the  software  production 
subsystem,  the  controlling  subsystem,  and  the  planning 
subsystem,  "The  model  was  developed  on  the  basis  of  a  battery 
of  27  field  interviews  of  software  project  managers  in  five 
software  producing  organizations,  supplemented  by  an  extensive 
database  of  empirical  findings  from  literature."  (Abdel-Hamid, 
1989)  The  human  resource  management  subsystem  accounts  for 
variables  related  to  the  workforce,  namely,  the  hiri.-.g  rate, 
training,  and  turnover  of  project  personnel.  The  software 
production  subsystem  models  the  designing,  coding,  and  testing 
phases  of  the  software  development  lifecycle.  This  subsystem 
also  accounts  for  the  quality  assurance  effort  required  for 
project  develop  as  well  as  the  actual  productivity  of  the 
project  team.  In  contrast  to  actual  productivity,  perceived 
productivity  is  described  in  the  control  subsystem.  Perceived 
productivity  directly  influences  a  manager's  estimate  of 
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Figure  1.  Model  Structure 
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project  tasks  perceived  to  be  completed.  This  estimate, 
however,  is  often  unrealistic  with  regard  to  software 
development  since  one  must  have  accurate  knowledge  of  rates  of 
accomplishment  and  resources  expended  to  date  (Abdel-Hamid  and 
Madnick,  1989).  Thus,  this  variable  often  is  no  more  than  a 
measurement  of  budgeted  resources  that  have  been  expended. 
The  planning  subsystem,  the  final  subsystem  of  the  model, 
provides  initial  project  estimates  such  as  project  cost, 
schedule,  and  staffing.  As  the  project  continues  through  its 
lifecycle,  these  estimates  will  change  to  reflect  management's 
decisions . 

C.  EXPERIMENTAL  DESIGN 

The  research  design  is  illustrated  in  Table  2.  The 
experiment  used  a  factorial  design  with  two  components  in 
order  to  capture  the  feedback  condition  and  the  project  type. 
These  components  are  between-subjects  and  within-subjects. 


TABLE  2.  EXPERIHEKrAL  DESIGN 


Type  of  information 

Coanltlve  Feedback  Outcoae  Feedback 

Feedforward 

Project  Ho. 

Project  No. 

Project  No. 

1  2 

3 

1  2 

3 

1 

2 

3 

Order 

Order 

1 

I  U 

F 

I  U 

F 

I 

U 

F 

Order 

2 

U  I 

F 

U  I 

F 

U 

I 

F 

Order 

3 

F  U 

I 

F  U 

I 

F 

U 

I 

Notes  1 

1  1. 

Participants  were 

randoaly  asaicned  to  one  of  the  feedback 

conditions  and 

one  of 

the  sequences  of 

task 

conditions. 

2. 

I,  U,  and  F  refer 

to 

ideal,  undersize,  and  fixedslza  projects,  respectively. 
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1.  Between-Sub jects 

The  fundamental  objective  driving  the  experiment  is  to 
determine  the  effect  of  cognitive  feedback,  i.e.,  determine 
how  best  the  operator  can  be  conveyed  a  model  of  the  system 
over  time  (Brehmer,  1987):  through  feedforward,  outcome 
feedback,  or  cognitive  feedback.  This  is  the  rationale  for 
the  between- subject  component. 

2.  Within-Sub jects 

In  addition  to  determining  if  systematic  differences 
exist  among  experimental  conditions,  feedback  studies  also 
seek  to  study  the  effect  within  each  condition  over  time. 
This  is  referred  to  as  within-subject  design  (Barlow  and 
Hersen,  1984,  p.  66)  and  involves  multiple  measurements  over 
different  points  in  time.  In  this  experiment,  the  within 
subjects  aspect  was  operationalized  by  using  three  separate 
projects,  namely,  the  ideal,  fixedsize/bad  estimate,  and 
undersize  projects. 

Randomization  between  and  within  subjects  was  achieved 
using  a  Latin  Square  Design  as  follows  (Kirk,  1982,  pp.  311- 
312) : 

First,  each  project  was  assigned  a  corresponding  letter. 

A:  Ideal  Project(IL) 

B:  Fixedsize/bad  estimate  Project(FB) 

C:  Undersize  Project(UN) 

Next,  two  sequences  of  random  numbers  were  generated. 

(2,1,3)  (3,1,2) 
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The  appropriate  square  is  then  selected. 

ABC 
B  C  A 

CAB 

Rows  are  then  ordered  according  to  the  first  set  of  random 
numbers . 

B  C  A 

ABC 
CAB 

Next,  columns  are  ordered  according  to  the  second  set  of 

random  numbers. 

ABC 
CAB 
B  C  A 

Finally,  the  notation  is  converted  according  to  project  name. 


G1 

G2 

G3 

IL 

FB 

UN 

Task 

UN 

IL 

FB 

Order 

FB 

UN 

IL 

Therefore,  Group  1  receives  the  projects  in  the  order:  ideal, 
undersize,  and  fixedsize/bad  estimate. 

D .  SUBJECTS 

The  experiment  was  conducted  using  56  graduate  students. 
Participants  were  divided  into  nine  groups  based  on  the 
feedback  condition  and  task  order.  Table  3  shows  the 
feedback  condition  and  task  order  provided  to  each  group. 

Each  subject  was  assigned  a  number  from  1  to  56  according 
to  the  alphabetical  order  of  his  last  name.  Two  digit  random 
numbers  were  then  generated  using  a  random  number  taUsle. 
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TABLE  S.  GROUP  ASSIGTMEKTS 


GROUP  NlitBER 


FEEDBACK  COtOmON/O'ASK  ORDER 


1-1 

1-2 

1- 3 

2- 1 
2-2 

2- 3 

3- 1 
3-2 
3-3 


Cognitive  Feedbacks  G1  Task  Order 
Cognitive  Feedback*  G2  Task  Order 
Cognitive  Feedback*  G3  Task  Order 
Outcome  Feedback*  61  Task  Order 
Outcome  Feedback*  G2  Task  Order 
Outcome  Feedback*  63  Task  Order 
Feedforward*  61  Task  Order 
Feedforward*  62  Task  Order 
Feedforward*  63  Task  Order 


Subjects  corresponding  to  the  first  six  numbers  were  assigned 
to  group  1-1,  the  next  six  in  group  1-2,  etc.  Duplicates  and 
random  numbers  greater  than  56  were  disregarded.  Due  to  the 
number  of  subjects  not  being  evenly  divisible  by  nine,  groups 
1-3  and  2-3  had  seven  subjects  each. 

1.  Participant  Profiles 

Two  types  of  subject  characteristics  could  potentially 
have  affected  results  in  this  experiment:  demographic  factors 
and  task- specific  factors.  Demographic  factors  were 
operationalized  as  age*  full  time  work  experience*  years  since 
completion  of  undergraduate  education*  familiarity  in  working 
with  computers  and  hours  per  week  a  subject  spent  working  on 
computers.  Table  4  profiles  the  subjects  with  respect  to 
demographic  factors.  The  task- specif ic  factor  was 
operationalized  by  asking  subjects  whether  they  had  any  prior 
experience  in  the  task.  It  was  determined  that  none  of  the 
subjects  had  any  significant  experience  in  software  project 
management . 
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TABLE  4.  PARTICIPAKT  PROFILES.  (Means  I. 


By  feedback  condition i 

Cognitive 

Outcome 

Feedback 

Feedback 

Feedforward 

AGE 

34 

33 

31 

HK  EXP 

14 

13 

10 

Y  U6RAD 

10 

10 

8 

FAH  COHP 

5 

6 

5 

HRS  COMP 

10 

9 

9 

By  project  order i 

Order  1 

Order  2 

Order  3 

AGE 

34 

30 

34 

HK  EXP 

14 

10 

13 

Y  U6RAD 

10 

8 

11 

FAM  COIP 

6 

5 

5 

HRS  conp 

12 

8 

9 

Key  I  AGE  =  A^e  of  subjects  (Years) 

(W_EXP  =  Full  tine  work  experience  (Years) 

Y_U6RA0  =  Years  since  coepletion  of  undergraduate  education 

FAM_CaMP  =  Faelllarity  of  subjects  with  coeputers  (1  =  not  fanlliarr  9  =  very  faniliar) 
HRS_CQHP  =  Hours  per  week  spent  using  computers 


E.  INFORMATION  PROVIDED  TO  SUBJECTS 

Subjects  were  provided  different  types  of  information 
based  upon  the  feedback  condition  corresponding  to  their 
group.  During  the  lifecycle  of  each  project,  the  experiment 
software  would  pause  at  40  day  intervals  to  allow  subjects  to 
review  this  information. 

1 .  Outcome  Feedback 

Subjects  receiving  outcome  feedback  were  given  only 
one  report,  the  Project  Status  Report,  at  the  end  of  each 
interval.  Information  provided  in  this  report  is  shown  in 


Table  5. 


TABLE  5.  PROJECT  STATUS  REPORT 


CURRENT  INTERVAL  STATISTICS i  Elapsed  Time  = 

80 

Days 

INITIAL  ESTDlATESi  (These  will  not  change  throughout 

the  project  i 

Project  size 

1.067 

Tasks 

Project  duration 

41  S 

Days 

REPORTED  STATISTICS  at  Tima  =====> 

80 

Days 

y.  Development  Reported  to  be  complete 

10.26 

Percent 

y.  Testing  Reported  to  be  complete 

0.00 

Percent 

Perceived  Total  Project  Size  at  this  point 

1  ,066.67 

Tasks 

Perceived  Total  Project  Cost  at  this  point 

5,721.00 

Man  Days 

Total  Number  -  Fulltime  E<iulvalent  Staff 

4.3 

Fulltime  staff 

New  Estimate  of  Project  Duration  (start  -  end) 

829 

Days 

Maximum  Tolerable  Completion  Date 

479 

Days 

Total  Man  Days  Expended 

548.36 

Man  Days 

Total  Numiser  of  Tasks  developed  to  date 

137.45 

Tasks 

PRESS  <EHTER>  TO  RETURN  TO  MENU 

2 .  Feedforward 

In  addition  to  receiving  the  Project  Status  Report 
(Table  5),  subjects  in  the  feedforward  groups  were  given  a 
separate  training  lecture  prior  to  the  experiment  which 
provided  further  insight  into  the  human  resource  management 
subsystem  of  the  Model  of  Software  Project  Management.  Figure 
2  is  an  exploded  view  of  this  subsystem. 

The  first  part  of  the  feedforward  training  provided 
subjects  with  instruction  on  two  concepts  critical  in  the 
human  resource  management  subsystem;  average  productivity  and 
net  cumulative  contribution.  To  demonstrate  the  importance  of 
carefully  considering  each  of  the  two  concepts,  the  following 
human  resource  management  problem  was  given: 

The  initial  project  team  consists  of  five  people  each 
with  a  productivity  of  ten  lines  of  code  (LOG)  per  man  day 
(MD)  thus  giving  a  total  output  of  50  LOG  per  day  for  the 
entire  team.  The  assumption  is  that  the  project  is  behind 
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Figure  2.  Human  Resource  Management  Subsystem 


schedule  and  the  manager  must  make  a  decision  either  to  add  a 
new  person  to  the  team  or  accept  the  schedule  slippage. 

Case  one  examines  the  effect  of  adding  a  new  person 
with  a  productivity  of  8  LOG  per  man  day.  If  this  person  is 
added,  it  is  expected  that  the  productivity  of  the  old  team 
will  decrease  by  10  percent  (i.e.,  9  LOC/MD)  due  to  training 
and  the  added  communication  overhead.  Case  two  also  adds  a 
new  person  but  this  person's  productivity  is  only  4  LOC  per 
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man  day.  Again,  this  will  cause  a  10  percent  decrease  in  the 
productivity  of  the  old  team. 

In  the  first  case  the  output  of  the  team  increases  to 
53  LOC/Day  (given  by  5  team  members  x  9  LOC/MD  +  8  LOC/MD  for 
the  new  member).  The  average  productivity  of  the  team  is  now 
8.8  LOG  (53  divided  by  6  team  members).  The  net  cumulative 
contribution  of  the  new  person  is  53-50  or  3  LOC/MD.  Since 
the  average  productivity  of  the  team  decreases,  the  cost  of 
the  project  will  increase,  but  since  the  net  cumulative 
contribution  of  the  new  person  is  positive,  the  schedule  will 
go  down . 

In  case  two,  the  output  of  the  team  is  only  49  LOC/Day 
(5  team  members  x  9  LOC/MD  +  4  LOC/MD  for  the  new  team 
member).  The  average  productivity  of  the  team  is  8.1  LOC  (49 
divided  by  6  team  members),  and  the  net  cumulative 
contribution  of  the  new  person  is  49-50  or  -1  LOC/MD!  Thus 
the  addition  of  the  new  team  member  is  detrimental  to  the 
project,  not  only  driving  up  the  project's  cost  but  its 
schedule  as  well. 

Although  the  mathematics  of  the  concepts  are 
relatively  simple,  the  importance  of  the  lesson  is  to  realize 
that  adding  a  person  (or  people)  to  a  late  project  will  not 
always  Improve  the  project's  schedule.  The  manager  must  look 
closely  at  the  average  productivity  of  the  project  team  as 
well  as  the  net  cumulative  contribution  of  any  team  members 
added . 
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The  second  part  of  the  feedforward  training  lecture 
focused  on  considerations  involved  in  the  willingness  to 
change  workforce  (WCWF) .  Subjects  were  presented  with  the 
equation, 

Workforce  =  Indicated  *  WCWF  +  Current  *  (1-WCWF) 
Sought  Workforce  Workforce 

and  its  relation  to  the  WCWF  curve  (Figure  3).^  The 

instructor  explained  to  subjects  that  a  manager  at  a  point  in 

the  project  which  yields  a  WCWF  value  of  zero  from  the  curve 

is,  in  essence,  not  willing  to  change  his  workforce  and  thus, 

the  workforce  sought  will  be  equal  to  the  current  workforce. 

If,  however  the  manager  is  at  a  point  in  the  project  where  the 

WCWF  is  one,  the  manager  is  very  willing  to  change  the 

workforce  and  thus,  the  workforce  sought  is  equal  to  the 

2 

indicated  workforce. 

3 .  Cognitive  Feedback 

Subjects  receiving  cognitive  feedback  also  received 
the  Project  Status  Report  (Table  5)  after  each  40  day 
interval.  Additionally,  personnel  receiving  cognitive 
feedback  had  the  option  to  view  a  Cognitive  Feedback  Report  as 
well  as  four  plots.  The  Cognitive  Feedback  Report  (Table  6) 


Subjects  were  told  that  the  time  parameter  referred  to  in 
the  figure  was  the  sum  of  two  parameters  from  SDM's  human  resource 
management  subsystem.  These  two  parameters  are  the  hiring  delay, 
set  at  30  days  for  this  experiment,  and  the  assimilation  or 
training  time,  set  at  20  days. 

2 

The  indicated  workforce  is  /nonymous  w  h  the  workfc  ce 
necessary  to  stay  on  schedule. 
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TABLE  6.  COGNITIVE  FEEDBACK  REPORT 


CURRENT  INTERVAL  STATISTICS!  Elapsed  Tiae  = 

80 

Days 

INITIAL  ESTIHATESi  (These  Hill  not  change  throughout  the  project  1 

Project  Size 

1,067 

Tasks 

Project  Duration 

41S 

Days 

REPORTED  STATISTICS  at  Tiae  =  a  =  x  =  s  x> 

Fraction  of  Horkforee  that  is  Experienced 

80 

0.9 

Days 

Perceived  Average  Productivity 

Coaaunieation  Overhead 

0.4 

0.01 

Tasks/Tlar 

i-day 

Total  Nuabar  -  Fulltlae  Equivalent  Staff 

4.5 

Fulltlae 

Staff 

Estlaated  Horkforee  Needed  to  Stay  on  Schedule 

4.5 

Fulltiae 

Staff 

PRESS  <EKrER>  TO  RETURN  TO  MENU 


provided  information  on  specific  workforce  variables.  Four 
plots,  described  as  the  Project  Size  Plot,  the  Staffing  and 
Schedule  Plot,  the  Workforce  Mix  Plot,  and  the  Workforce 
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Productivity  Plot,  were  provided  to  assist  subjects  in 
spotting  trends  developing  throughout  the  project  lifecycle. 

The  plot  on  Project  Size  (Figure  4)  plotted  two 
variables,  the  perceived  total  project  size  to  date  (PJBSZ) 
and  the  perceived  total  project  cost  to  date  (JBSZMD),  over 
the  project  lifecycle.  This  plot  provided  subjects 


information  on  whether  any  schedule  or  budget  slippage  was  a 
result  of  either  unexpected  increases  in  the  project's  size 
(e.g.,  due  to  changes  in  users'  requirements),  or  that  the 
effort  required  to  complete  the  project  was  initially 


Figure  4.  Project  Size  Plot 
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underestimated  (e  q.  ,  because  the  project  complexity  was 
underestimated) .  If  the  former  case  were  true,  subjects 
would  expect  to  see  the  variable  PJBSZ  increase  over  time.  If 
the  latter  were  true,  then  the  variable  JBSZMD  would  increase 
over  time. 

The  plot  on  Project  Staffing  and  Schedule  (Figure  5) 
plotted  three  variables  of  the  project  lifecycle:  the  total 
number  of  fulltime  equivalent  staff  (FTEQWF),  the  new 


estimate  of  project  duration  from  start  to  end  (SCHCDT),  and 
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the  estimated  workforce  needed  to  stay  on  schedule  (WFINDC). 
This  plot  provided  feedback  on  the  trade-off  between 
minimizing  schedule  over-runs  versus  minimizing  cost  over¬ 
runs.  When  a  project  runs  into  difficulties,  a  manager  can 
choose  to  stick  with  the  project's  schedule  (SCHCDT)  by 
increasing  the  workforce  level  (FTEQWF).  This  practice  always 
increases  the  cost  of  a  project.  On  the  other  hand,  a  manager 
might  wish  to  minimize  his/her  cost  over-run,  by  avoiding  an 
increase  in  the  workforce  level,  and  instead,  opt  to  increase 
the  project's  scheduled  completion  date.  The  indicated 
workforce  level  (WFINDC)  is  provided  as  an  estimate  of  the 
workforce  needed  to  stay  on  schedule. 

The  plot  on  Workforce  Mix  (Figure  6)  provided  subjects 
with  feedback  on  their  staffing  decisions.  In  general, 
staffing  decisions  have  the  greatest  impact  on  productivity. 
This  option  plotted  three  variables:  the  total  number  of 
fulltime  equivalent  staff  (FTEQWF),  the  fraction  of  the 
workforce  that  is  experienced  (FRWFEX),  and  the  planned 
workforce  (PLANWF)  over  the  project  lifecycle.  This  plot  was 
deemed  useful  for  two  reasons.  First,  larger  workforces  will, 
in  most  cases,  be  less  productive  because  of  the  increases  in 
communication  and  training  overheads.  Second,  the  workforce 
mix  (i.e.,  percent  of  experienced  vs  new  staff  in  the 
workforce)  will  also  have  an  impact  on  productivity.  The 
larger  the  percentage  of  experienced  people  in  the  workforce, 
the  more  productive  the  workforce. 
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Time  {Days)  Der'od 

Figure  6.  Workforce  Mix  Plot 

The  plot  on  Workforce  Productivity  (Figure  7)  provided 
subjects  with  information  on  the  average  productivity  of  their 
team.  It  displayed  the  relationship  between  the  total  number 
of  fulltime  eqpaivalent  staff  (FTEQWF),  the  perceived  average 
productivity  of  the  workforce  (ASSPRD),  and  the  communication 
overhead  associated  with  the  workforce  (COMMOH)  throughout  the 
project  lifecycle.  The  basis  for  presenting  this  plot  was, 
again,  twofold.  First,  the  larger  the  workforce,  the  lower 
the  average  productivity.  This  is  due  to  the  time  people 
"waste"  in  communicating  with  teammates.  Second,  the 
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communication  overhead  curve  shows  subjects  exactly  what 
percentage  of  a  person's  time  is  wasted  in  communication  with 
others . 


F .  EXPERIMENTAL  SETTING 

The  experiment  was  conducted  in  two  computer  labs  with  two 
attendants  per  lab.  Each  subject  was  assigned  a  specific 
terminal  and  was  given  documentation  (see  Appendix)  and  a  disk 
according  to  his/her  group  assignment.  Subjects  were  given 
time  to  read  over  the  documentation  and  ask  questions  about 


the  conduct  of  the  experiment.  Questions  pertaining  to  the 
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task  were  not  permitted.  After  reading  the  documentation, 
subjects  ran  several  intervals  of  a  "dummy"  project  to 
familiarize  them  with  the  reports,  plots  and  keystrokes 
required  to  traverse  through  the  screens  they  were  presented 
with.  Attendants  assisted  the  subjects  with  any  technical 
difficulties.  After  completing  the  trial  run,  subjects  were 
permitted  to  proceed  with  each  of  the  three  projects  at  their 
own  pace. 

G .  DEPENDENT  MEASURES 

Two  dependent  variables,  deviation  from  initial  estimates 
and  staff  productivity,  were  used  to  analyze  the  performance 
of  each  subject.  Two  numbers  were  used  to  determine  the 
deviation  from  initial  estimates.  The  first  number  was  the 
difference  between  the  subject's  completion  time  and  the 
estimated  completion  time.  The  second  number  was  the 
difference  between  the  subject's  final  cost  and  the  estimated 
project  cost.  These  numbers  were  then  averaged  to  yield  the 
deviation  (overrun  or  underrun)  from  the  initial  project 
estimates.  Project  time  is  defined  as  the  length,  measured  in 
days,  required  for  the  subject  to  successfully  manage  each 
project  from  start  to  end.  Project  cost  measures  the 
resources  expended,  in  man  days,  to  complete  each  project. 
The  staff  productivity  is  defined  as  tasks  per  man-day  and  was 
determined  by  dividing  the  number  of  tasks  associated  with 


each  project  by  the  total  cost  of  that  project  len  managed  by 
a  particular  subject. 

Lower  benchmarks  for  time  and  cost  were  determined  for 
each  of  the  three  projects  by  running  15  simulations  for  each 
project  using  random  staff  sizes.  Minimum  and  maximum  staff 
sizes  for  each  project  type  were  determined  from  subjects' 
decisions.  Five  hundred  random  numbers  were  then  generated 
for  each  project  to  fall  within  the  minimum  and  maximum  staff 
size.  The  15  simulations  were  then  run  for  each  project  using 
staff  sizes  taken  sequentially  from  the  500  generated  in  that 
project's  staff  range. 

H.  EXPERIMENTAL  RESULTS 

Tables  7  through  11  summarize  the  results  of  the 
experiment.  Cases  in  which  subjects  made  significant  errors 
were  discarded,  and  data  from  45  subjects  was  retained  for 
analysis.  Table  7  shows  performance  between  subjects,  as  well 
as  within  subjects,  with  respect  to  deviations  from  the 
initial  project  estimates. 

Applying  the  approach  suggested  by  Winer  (1971,  p.  697), 
the  following  analysis  of  variance  (ANOVA)  model  suited  for 
multiple  Latin  Squares  was  used  to  test  Hypotheses: 

Yijk(l)n>  =  M  +  +  83  +  +  Xi  +  + 

(yx),fl  +  (ya),,j  +  (yB)„3  +  where: 

^  is  constant, 

Oj  is  the  sequence  of  the  projects  (i  =  1,2,3), 
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Sj  is  the  order  of  the  project  (j  =  1,2,3), 


is  the  feedback  condition  (k  =  1,2,3,  where  1  = 
cognitive  feedback,  2  =  feedforward,  and  3  =  outcome 
feedback) , 

Xj  is  the  type  of  project  (1  =  1,2,3,  where  1  =  ideal, 

2  =  undersize,  and  3  =  f ixedsize/bad  estimate), 

5|„  is  the  experimental  participant  (m  =  1 ,  .  .  .  ,  45 ) , 
ejj]j(j)„  is  the  experimental  error  term. 


TABLE  7.  DEVIATIONS 

FROM  INITIAL 

ESTIMATES.  Keans 

and  ( Standard 

Deviations  ). 

N 

Ideal 

Undersize 

Fixadsize 

Cognitive  Feedback 

15 

0.448 

11.501 

39.  Z9 

(1.7Z5I 

(4.43) 

1 1 .854) 

Feedforward 

15 

3.858 

17.484 

47. Z1 

(1.794) 

( Z.058) 

(5.605) 

Outcowe  Feedback 

15 

10.014 

Z5.753 

57.0Z7 

(5.710) 

(6.05Z) 

1 7.041 ) 

Random  Baseline 

15 

11. IZ 

Z7.98 

56.  ZZ 

(6.1Z) 

(7.Z3) 

(9.1Z) 

The  analysis  was  conducted  with  the  General  Linear  Models 
procedure  (SAS,  1987).  Table  8  contains  the  ANOVA  results. 
The  results  show  that  the  performance  of  subjects  across  the 
three  different  feedback  conditions  was  significantly 
different  (F=345.89,  p=0.0001).  The  null  hypothesis  of  no 
significant  differences  among  feedback  conditions  is  therefore 
rejected.  In  other  words,  the  results  indicate  that  the 
subjects'  performance  was  influenced  by  the  type  of 
information  provided  to  them. 
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TABLE  8.  AMALTSIS 

OF  VARIANCE. 

Dependent  Variablei  Deviation 

from  Initial 

Estimates. 

Source  of 

Variance 

S.S. 

Degrees  of 
freedom 

F-Value 

P 

R-Square 

Model 

50769.00 

57 

136.54 

0.  301 

0.89 

Type  of 

Information 

4530.01 

2 

345.89 

0.0001 

Type  of  Task 

44166.85 

2 

3372.37 

0.0001 

Sequence 

16.53 

2 

1.26 

0.2887 

Order 

3.16 

2 

0.24 

0.7862 

Participant 

2041.19 

41 

7.60 

0.0001 

GroupKTask 

4.48 

4 

0.23 

0.8126 

GroupaOrder 

6.67 

4 

0.26 

0.9036 

Mithin  Groups 

504.22 

77 

Additionally,  Table  8  shows  that  the  performance  within- 
subjects  was  significantly  different  depending  on  the  type  of 
project  confronting  them  (F=3372.37,  p=0.0001).  The  null 
hypothesis  of  no  significant  differences  in  performance 
depending  on  type  of  project  presented  is  therefore  also 
rejected. 

The  Tukey  Test  for  Additivity  indicated  no  presence  of 
interaction  effects  between  the  sequence  of  projects  (p>0.2), 
or  the  order  in  which  projects  were  completed  (p>0.7).  Also, 
there  were  no  interaction  effects  between  the  type  of 
information  provided  and  the  type  of  project  (p>0.8),  nor  the 
type  of  information  provided  and  the  order  of  project  (p>0.9). 

Table  9  summarizes  tests  comparing  results  from  each  of 
the  feedback  conditions  with  the  random  baseline  with  respect 
to  deviations  from  initial  estimates.  The  mean  total 
deviation  for  subjects  in  the  cognitive  feedback  and 
feedforward  conditions  was  lower  than  the  random  baseline  in 
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TABLE  9.  COMPARISON  BETWEEN  EXPERIMENTAL  GROUPS  AM)  BASELINE.  Dependent  Variable i 
Deviation  fro*  Initial  Estimates. 


Comparison 

Ideal 

Total 

deviation 

Undersize 

Total 

deviation 

Fixedsize 

Total 

deviation 

Cognitive  Feedback 

r>ef  b 

r>cfb 

r>cf  b 

Feedforward 

r>ff 

r>ff 

r>ff 

Outcome  Feedback 

n.a. 

n.s. 

n.s. 

Notesi  1.  cfbi  cojrnitive  feedback}  ffi  feedforward}  n.s.i  not  significant}  ri  random 
baseline . 

Z.  The  comparisons  represent  one-tailed  (directional)  t-tests  performed  on  the 
means.  ThuS}  r>cfb  indicates  that  the  mean  for  that  variable  in  the  random 
baseline  was  higher  than  the  mean  in  the  cfb  condition}  at  p<0.05.  n.s. 
indicates  that  differences  in  the  means }  if  any}  were  not  significant  at 
p<0.05. 


all  three  projects.  Subjects  receiving  only  outcome  feedback 
showed  no  significant  performance  differences  from  the  random 
baseline  in  any  of  the  three  projects. 

Table  10  summarizes  staff  productivity  results.  This 
information  shows  the  actual  productivity  in  tasks/man-day  of 
the  simulated  staff  as  managed  by  the  subject.  Table  11 
contains  the  ANOVA  results  for  the  staff  productivity  data. 
Again,  there  was  a  significant  difference  in  performance 
between  subjects  (F=106.88,  p=0.0001)  as  well  as  within- 
subjects  (F=109.59,  p=0.0001).  The  Tukey  Test  for  Additivity 
indicated  no  presence  of  interaction  effects  between  the 
sequence  of  projects  (p>0.9),  nor  the  order  of  the  projects 
(p>0.7).  Also,  no  interaction  effects  were  present  between 
the  type  of  information  provided  and  the  type  of  project 
(p>0.6),  nor  the  type  of  information  provided  and  the  order  of 
projects  (p>0.4). 
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TABLE  10.  STAFF  PRODUCTIVITY.  Heana  and  {Standard  Deviations  I . 


M 

Ideal 

Undersize 

Fixedsize 

Cognitive  Feedback 

15 

0.3V3 

0.307 

0.251 

(0.051  ) 

(0.047) 

(0.051 ) 

Feedforward 

15 

0.317 

0.249 

0.182 

( 0 . 049 1 

(0.051 ) 

(0.049) 

Outcone  Feedback 

15 

0.249 

0.177 

0.112 

(0.051  1 

(0.057) 

(0.037) 

Random  Baseline 

15 

0.238 

o.iao 

0.131 

(0.0601 

(0.071 ) 

(0.032) 

TABLE  11.  AHALYSIS  OF  VARIANCE 

Dependent  Variable ■  Staff 

Productivity . 

Source  of 

Degrees  of 

Variance 

S.S. 

freedom 

F-Value 

P 

R-Square 

Model 

1.62 

57 

8.89 

0.0001 

0.86 

Type  of 
Information 

0.43 

2 

106.88 

0.0001 

Type  of  Task 

0.44 

2 

109.59 

0.0001 

Seijuance 

0.002 

2 

0.06 

0.9443 

Order 

0.006 

2 

0.09 

0.7869 

Participant 

0.71 

41 

1.37 

0.0401 

GroupcTaak 

0.005 

4 

0.66 

0.6234 

6roup«0rder 

0.007 

4 

0.98 

0.4235 

Nlthln  Groups 

0.16 

77 

i 
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IV.  CONCLUSIONS 


A.  SUMMARY  OF  RESULTS 

The  objective  of  this  study  was  to  investigate  the  effects 
of  cognitive  feedback,  outcome  feedback,  and  feedforward  on 
decision  makers  in  a  dynamic  decision  environment  such  as 
software  project  management.  Chapter  II  (section  B.2)  pointed 
out  that  past  research  has  shown  the  dysfunctional  effects  of 
outcome  feedback.  These  dysfunctional  effects  have  led 
researchers  to  seek  alternative  means  to  iir.prov/e  decision 
quality.  Chapter  II  (section  C)  discusses  two  alternatives  to 
outcome  feedback:  cognitive  feedback  and  feedforward.  Both 
cognitive  feedback  and  feedforward  seek  to  assist  the  decision 
maker  in  formulating  a  "mental  model"  of  the  task  which 
confronts  him/her.  Without  this  model,  decision  makers  have 
exhibited  poor  performance  in  handling  the  delays  and 
oscillations  associated  with  complex  dynamic  systems. 
Additionally,  Chapter  II  (section  C.3)  explains  why  one  would 
expect  cognitive  feedback  to  improve  decision  makers' 
performance  more  than  feedforward.  The  results  of  this  study 
support  these  past  findings.  As  the  analysis  of  variance 
tests  showed,  there  was  a  significant  difference  in  subjects' 
performance  depending  on  the  type  of  feedback  with  which  they 
were  provided.  Subjects  in  the  cognitive  feedback  condition 
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experienced  less  deviations  from  the  initial  project  estimates 
than  subjects  in  the  feedforward  or  outcome  feedback 
condition.  Additionally,  staffs  managed  by  subjects  receiving 
cognitive  feedback  showed  greater  productivity  than  staffs 
managed  by  subjects  in  the  other  feedback  conditions.  Also, 
tests  comparing  feedback  groups  with  the  random  baseline 
showed  that  subjects  in  the  cognitive  feedback  and  feedforward 
conditions  performed  better  than  the  random  baseline  in  each 
of  the  three  projects,  whereas  there  was  no  significant 
performance  difference  between  subjects  receiving  outcome 
feedback  and  the  random  baseline. 

B.  FEEDBACK  AS  A  DECISION  TOOL 

The  results  of  this  experiment  provide  several 

implications  for  the  design  of  project  control  systems, 

specifically  those  in  support  of  software  project  managers. 

As  Brehmer  (1987)  concluded  from  his  DESSY  experiment, 

These  results  show  that  system  designers  cannot  rely  upon 
the  operators  to  develop  good  mental  models  f  complex 
systems.  This  Implies  that  we  need  to  develop  .means  that 
help  the  subjects  develop  such  models,  or  possibly  means 
that  eliminate  the  need  for  predictive  models,  (p.  30) 

Brehmer  provides  several  suggestions  for  improving  the  quality 

of  information  systems.  The  most  important  one,  with  respect 

to  this  study,  states  the  need  to  communicate  information 

about  the  system  in  nonverbal  form.  As  this  study  showed, 

presenting  information  in  graphical  form  did,  in  fact,  raise 

the  performance  level  of  subjects  receiving  that  information. 
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C.  LIMITING  FACTORS  TO  GENERAL I Z ABILITY^ 

Although  a  majority  of  the  subjects  in  this  experiment  had 
some  managerial  experience,  the  question  remains  whether  they 
had  enough  experience  to  play  the  game.  In  other  words,  is  it 
reasonable  to  make  a  comparison  between  their  performance  and 
that  of  real  life  software  managers? 

Remus  (1986)  found,  in  a  study  to  investigate  the  use  of 
graduate  students  as  surrogates  for  managers,  that  no 
significant  differences  existed  between  the  students  and 
managers  in  making  production  scheduling  decisions.  Although 
software  project  management  is  somewhat  different  from  the 
task  presented  in  Remus'  experiment,  it  is  similar  enough  to 
apply  his  findings  and  assume  that  software  engineering 
graduate  students  are  reasonable  surrogates  in  this 
experimental  investigation. 

The  next  limiting  factor  to  consider  is  the  nature  of  the 
particular  project  environment.  One  should  take  caution 
against  generalizing  the  results  presented  in  Chapter  III  to 
all  types  of  project  situations.  In  this  experiment,  each  of 
the  three  projects  was  developed  in  a  familiar  in-house 
environment  i.e.,  what  is  typically  described  as  an  organic- 
type  project  environment  (Boehm,  1981,  pp.  78-82). 


3 

This  section  is  based  on  an  unpublished  paper  by  Abdel-Hamid, 
Sengupta,  and  Ronan  (1990)  which  describes  a  similar  experiment 
using  the  SDM  gaming  interface. 
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Finally,  it  is  difficult  to  claim  external  validity  for 
laboratory-type  studies.  A  review  of  the  gaming  literature  by 
Remus  (1978)  does,  however,  indicate  considerable  similarity 
between  decision  making  in  games  and  managerial  decision 
making  per  se .  Since  the  project  games  in  this  experiment  are 
a  simulation  of  three  real  life  software  projects,  there 
seems  to  be  no  reason  why  there  should  be  any  exception  to 
Remus'  general  findings. 

D .  FUTURE  RESEARCH 

Based  on  the  discussion  in  Section  C  above,  one  particular 
path  for  future  research  using  the  SDM  gaming  interface  to 
investigate  managerial  decision  making  is  evident.  The  above 
experiment  could  be  replicated  using  real  software  project 
managers  as  the  subjects.  Although  using  graduate  students  as 
surrogates  in  research  studies  is  useful,  analyzing  the 
behavior  of  experienced  project  managers  could  provide  more 
significant  and  .oteworthy  results. 


42 


APPENDIX 


EXPERIMENTAL  INSTRUCTIONS  PROVIDED  TO  SUBJECTS 
A1  Written  description  given  to  subjects 
Introduction 

The  exercise  you  are  about  to  undertake  is  similar  in  many 
ways  to  flight  simulators  that  pilots  use  to  mimic  flying  an 
aircraft  from  takeoff  at  point  A  to  landing  at  point  B. 
Instead  of  flying  the  aircraft,  though,  this  simulator  mimics 
the  life  of  a  real  software  project  from  the  start  of  the 
design  phase  until  the  end  of  testing.  In  this  simulation, 
you  will  be  more  than  an  observer.  In  fact  you  will  play  a 
real  role  on  the  project:  that  of  the  project  manager. 

Specifically,  your  role  will  be  to  track  the  project's 
progress  using  a  number  of  reports  that  will  be  produced  for 
you  at  different  intervals  during  the  project.  You  will  then 
make  the  project's  staffing  decisions  based  on  the  knowledge 
you  gain  from  these  reports.  As  the  project  manager,  you  can 
hire  additional  staff  or  decrease  the  staffing  level  as  you 
deem  necessary  to  complete  the  project.  Your  objective  (like 
that  of  any  software  project  manager)  is  to  manage  your 
resources  wisely  and  efficiently  while  always  aiming  to  finish 
the  project  within  budget  and  on  schedule  (plus  any  safety 
factor  period  available.) 


Projects 

You  will  be  given  three  projects  to  manage,  all  of  them  real 
projects  conducted  in  a  real  organization.  The  organization 
is  on  the  leading  edge  in  its  software  engineering  practices. 
For  each  project,  you  will  be  given  a  project  profile 
containing  the  following  initial  information: 

Estimated  Project  Size(in  No.  of  tasks) 

Estimated  Schedule  Duration(in  No.  of  Work  Days) 

Estimated  Project  Cost (in  No.  of  Man  Days) 

Maximum  Toleradsle  Project  Duration(in  No.  of  Work  Days). 
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Your  Task 


Your  task  is  to  use  the  reports  generated  by  the  project  team 
at  different  points  in  the  project  to  determine  a  desired 
staffing  level  for  the  remainder  of  the  project.  Your 
objective  in  setting  the  staffing  level  should  be  to  decide  on 
the  best  compromise  between  finishing  on  an  acceptable 
schedule  while  avoiding  an  excessive  cost  overrun. 
Specifically,  you  should  try  to: 

(a)  complete  the  project  on  schedule, 

(b)  at  the  lowest  possible  cost,  and 

(c)  in  any  case,  complete  it  before  the  maximum  tolerable 
completion  date. 

Your  grade  for  the  simulation  will  be  based  on  an  equal 
weighting  of  two  factors: 

(a)  The  percentage  by  which  the  project  overshoots  the 

original  schedule.  Thus,  if  the  scheduled  completion  date 
for  the  project  is  200  days,  and  your  actual  completion 
date  is  240  days,  you  will  be  considered  to  have  overshot 
the  schedule  by  ( 240-200 )/200  =  20%. 

(b)  The  percentage  by  which  the  project  overshoots  the 

original  cost  estimate.  If  the  original  cost  estimate  is 
2,000  man  days  and  the  actual  cost  of  completion  is  2,500 
man  days,  you  will  be  considered  to  have  overshot  the  cost 
by  (2, 500-2, 000 )/2. 000  =  25%. 

The  following  are  some  important  things  to  consider  in  making 
your  decisions: 

1.  As  the  software  project  manager,  you  specify  the  desired 
staffing  level.  The  actual  staffing  level  may,  of  course, 
be  different  due  to  things  you  cannot  control  such  as 
turnover  and  lengthy  hiring  delays. 

2.  Each  project  is  initialized  with  a  particular  core  team  of 
full  time  equivalent  personnel  (FTE).  This  is  to  reflect 
that  fact  that  moat  projects  start  out  with  a  small  core 
team  of  personnel.  For  example,  project  2  may  be 
initialized  to  an  FTE  of  1.5. 

3.  The  hiring  delay  for  new  employees  can  take  up  to  6  weeks. 
Once  new  people  are  hired,  the  assimilation  period  for  a 
newly  hired  employee  is  typically  one  month  long.  This  is 
the  time  needed  to  train  a  new  employee  in  the  mechanics 
of  the  project  and  bring  him/her  up  to  speed.  A  new 
employee  (i.e.  one  that  is  being  trained)  is  only  half  as 
productive  as  an  experienced  employee. 
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4.  The  personnel  turnover  rate  is  20%  per  year. 


5.  At  different  points  in  the  project  you  will  be  given 
reported  information  on  the  status  of  the  project.  Two 
key  pieces  of  information  for  this  staffing  task  are:  (1) 
The  updated  estimate  of  the  total  project  cost  in  man  days 
(this  update  can  change  to  reflect  the  addition  of  new 
requirements  and/or  changes  in  the  estimate  of  the  team's 
overall  productivity);  and  (2)  Effort  expenditures  to  date 
(also  in  man  days).  Subtracting  the  second  from  the  first 
yields  the  "Remaining  Effort  in  man  days." 

6.  Let  us  say  that  at  some  point  in  the  project  the 
"Remaining  Effort"  is  1000  man  days,  the  remaining  time  is 
100  days  and  you  have  7  full  time  equivalent  employees 
working.  You  are,  thus,  in  a  position  where  you  ha/e  to 
use  your  judgement  to  do  one  of  the  following: 

1.  Stick  with  the  current  schedule.  If  so  then  you  will 
need  a  staff  size  of  1000/100  =  10  full  time 
employees . 

2.  Stick  with  your  staff  size  of  7.  This  means  the 
schedule  has  to  be  pushed  back.  In  this  case  the 
model  will  make  the  appropriate  adjustment  to  the 
schedule  for  you.  That  is  extend  it  to  1000/7  =  143 
man  days. 

3.  Do  a  bit  of  both.  That  is  increase  the  staff  size  a 
bit,  say  to  8,  which  will  also  mean  that  the  schedule 
will  be  extended  (appropriately  by  the  model)  to 
1000/8  =125  days. 


How  to  Play  the  Game 

1.  First,  take  some  time  to  practice  and  get  familiar  with 
the  system. 

(a)  Type  DUMMY  for  running  a  dvimmy  project. 

(b)  Run  the  dummy  project  for  1  interval. 

(c)  Go  through  all  the  options  in  the  menu.  Please  be 
sure  you  understand  all  of  them. 

(d)  When  you  are  done,  hit  <ESC>. 
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2.  The  real  simulation  starts  now.  You  will  be  given  three 

projects,  one  at  a  time.  When  you  are  done  with  one 

project,  you  can  move  on  to  the  next,  till  you  have 

completed  all  three  projects. 

3.  For  each  project; 

(a)  Insert  the  disk  you  are  given,  and  enter  a  command 
from  the  A>  prompt.  The  command  for  project  1  is 
PROJECTl ,  and  so  on. 

(b)  The  system  will  show  you  the  size  of  the  initial  core 
team  of  senior  designers  (the  full  time  equivalent 
number).  It  will  then  ask  you  for  your  initial 
desired  staffing  level. 

(c)  Next  it  will  run  through  the  first  simulation  time 
period  and  show  you  the  current  reported  statistics. 
Make  your  change  to  the  full  time  equivalent  staffing 
level  on  the  documentation  sheet  provided  after 
viewing  the  report. 

(d)  Perform  step  (c)  for  as  many  intervals  as  necessary, 
till  the  project  is  complete.  A  project  is  considered 
complete  when  there  are  less  than  40  days  left  for  the 
project  to  be  completed.  That  is, 

(New  Estimate  of  Project  Duration  -  Elapsed  Time)  <  40 
days . 

Thus,  if  the  New  Estimate  of  Duration  =  426  days,  and 
the  Elapsed  Time  =  400  days,  the  project  is  considered 
complete . 

(e)  There  is  no  need  to  turn  in  the  documentation  sheet 
after  each  interval  of  a  project.  However,  A  LAB 
ATTENDANT  MUST  VERIFY  YOUR  FINAL  RESULTS  at  the 
completion  of  each  project.  Call  a  lab  attendant  as 
soon  as  you  are  done  with  a  project. 

(f)  Complete  the  appropriate  questionnaire  in  your 
instruction  booklet. 

(g)  Move  on  to  the  next  project. 


Rules  of  the  Game 

*  You  will  be  required  to  provide  the  new  desired  staffing 
level  for  the  project  at  the  beginning  of  every  two-month 
interval  (consisting  of  40  work  days).  The  simulation 
will  stop  to  show  current  reported  statistics  and  accept 
a  desired  staffing  level  after  each  40  day  work  period. 
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Annotate  your  desired  staffing  level  on  the  documentation 
sheet  and  then  enter  it  at  the  simulation  prompt. 

YOU  MUST  WORK  ALONE.  You  are  not  allowed  to  discuss  this 
exercise  with  anyone  other  than  a  lab  attendant.  Also, 
please  refrain  from  discussing  this  with  any  member  in  the 
other  class  until  they  have  completed  the  exercise. 


Please  follow  the  guidelines  strictly.  The  system 
prompts,  along  with  instructions  in  this  booklet,  will 
guide  you  at  every  stage. 

If  you  are  in  doubt  about  what  to  do  next,  ask  for  a  lab 
attendant . 


Al.l  Further  instructions  provided  to  cognitive  feedback 
subjects 

How  to  use  and  Interpret  the  Plots 

Throughout  the  life  of  each  project  (starting  with  time 
elapsed  =  40  days),  you  will  have  access  to  a  series  of  plots 
providing  information  on  the  project.  As  the  project 
progresses  over  time,  the  plots  will  be  increasingly  more 
meaningful  in  making  your  staffing  decision.  This  and  the 
next  page  explain  how  to  interpret  and  use  the  plots  in  making 
your  decisions. 


Plot  on  Project  Size 

(refer  to  Chapter  III,  Figure  4) 

The  figure  below  provides  information  on  whether  any  schedule 
or  budget  slippage  is  a  result  of;  (1)  unexpected  increases  in 
the  project's  size  (e.g.,  due  to  changes  in  users' 
requirements),  or  (2)  that  we  initially  underestimated  the 
effort  required  to  complete  the  project  (e.g.  because  we 
underestimated  its  complexity).  In  the  first  case,  the 
Perceived  Total  Project  Size  (PJBSZ)  will  increase  over  time. 
In  the  second  case,  the  Perceived  Total  Project  Cost  (JBSZMD) 
will  increase  over  time. 
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Plot  on  Project  Staffing  vs  Schedule 
(refer  to  Chapter  III,  Figure  5) 

The  figure  below  provides  feedback  on  the  trade-off  between 
minimizing  schedule  over-runs  versus  minimizing  cost 
over- runs.  When  a  project  runs  into  trouble,  a  manager  can 
choose  to  stick  with  the  project's  schedule  (SCHCDT)  by 
increasing  the  workforce  level  (FTEQWF).  This  always 
increases  the  cost  of  the  project.  On  the  other  hand,  a 
manager  might  desire  to  minimize  his/her  cost  over-run,  by 
avoiding  an  increase  in  the  workforce  level,  and  instead, 
increase  the  project's  scheduled  completion  date.  The 
indicated  workforce  level  (WFINDC)  is  an  estimate  of  the 
workforce  needed  to  stay  on  schedule. 


Plot  on  Workforce  Mix 

(refer  to  Chapter  III ,  Figure  6) 

A  good  feedback  on  why  your  costs  may  be  higher  than  expected 
is  to  evaluate  your  staffing  decision.  In  general,  staffing 
decisions  have  the  greatest  impact  on  productivity.  First,  a 
larger  workforce  (FTEQWF)  will,  in  most  cases,  be  less 
productive  because  of  the  increases  in  communication  and 
training  overheads.  Second,  the  workforce  mix  (i.e.,  percent 
of  experienced  vs  new  staff  in  the  workforce)  will  also  have 
an  impact.  The  larger  the  percentage  of  experienced  people  in 
the  workforce  (FRWFEX),  the  more  productive  the  workforce. 


Plot  on  Workforce  Productivity 
(refer  to  Chapter  III,  Figure  1) 

This  figure  provides  information  on  the  average  productivity 
of  the  team.  In  general,  your  staffing  decisions  will  affect 
productivity  in  two  ways.  The  larger  the  workforce  you 
assemble  (FTEQWF),  the  lower  the  average  productivity 
(ASSPRD).  This  is  because  in  a  larger  workforce  people 
"waste"  more  time  communicating  with  team  mates.  This 
communication  overhead  is  plotted  above.  The  communication 
overhead  (COMMOH)  curve  tells  you  the  percentage  of  a  person's 
time  that  is  wasted  (on  average)  in  communication  with  others. 


***  You  are  now  ready  to  Start  PROJECT  1 


48 


^2  Information  provided  to  subject  for  first  project  -  order 
of  projects  varied  depending  on  subject' s  group  assignment 
(the  order  presented  here  is  the  same  as  the  sequence 
given  to  subjects  in  Group  1:  Ideal,  Undersize ,  and 
then  Fixedsize/bad  estimate) 

PROJECT  1 


Management's  Initial  Project  Estimates 

Initial  Estimate  of  Project  Size: 
Initial  Estimate  of  Project  Cost: 
Initial  Estimate  of  Project  Duration: 
Maximum  Tolerable  Project  Duration: 


1,067  Tasks 
3,721  Man  Days 
413  Days 
479  Days 


A  project  is  considered  complete  when  there  are  less  than  40 
days  left  for  the  project  to  be  completed.  That  is,  (New 
Estimate  of  Project  Duration  -  Elapsed  Time)  <  40  days. 


Staffing  Level  Sought  (in  FTE) 

Please  enter  your  staffing  decisions,  i.e.,(In  Full  Time 
Equivalent)  below: 

Initial  Estimate:  _ 


Time 

elapsed 

- 

40  days: _ 

Time 

elapsed 

- 

80  days: _ 

Time 

elapsed 

- 

120 

days:_ 

Time 

elapsed 

- 

160 

days :_ 

Time 

elapsed 

- 

200 

days :_ 

Time 

elapsed 

- 

240 

days:_ 

Time 

elapsed 

- 

280 

days:_ 

Time 

elapsed 

- 

320 

days:_ 

Time 

elapsed 

- 

360 

days:_ 

Time 

elapsed 

- 

400 

days 

Time 

elapsed 

- 

440 

day8:_ 

Time 

elapsed 

- 

480 

days:_ 

Time 

elapsed 

- 

520 

days : _ 
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Time  elapsed  -  560  days: 
Time  elapsed  -  600  days: 
Time  elapsed  -  640  days: 


***  WHEN  YOU  ARE  DONE,  PLEASE  CALL  FOR  A  LAB  ATTENDANT  *** 

AJ  Questions  answered  by  all  subjects  after  completion  of 
first  project 

1.  Describe  (in  words,  numbers,  equation,  etc)  what  decision 
rule  you  followed  in  deciding  on  the  staffing  level  in 
this  project: 


2.  Please  try  to  elaborate  on  the  thinking  process  you  went 
through  in  making  your  decisions  in  this  project  (use  back 
of  page  if  necessary): 


3.  How  clear  were  the  instructions  regarding  the  task? 


12  3 

Not  at  all 
Clear 


4  5  6  7 


8  9 

Very 
Clear 


4.  To  what  extent  was  the  report  on  the  progress  of  the 
project  helpful  in  Improving  your  own  decision? 

123456789 
Not  at  all  Very 

Helpful  Helpful 
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A3.1  Additional  question  asked  of  subjects  in  feedforward 
condition  only 

5.  To  what  extent  was  the  training  provided  before  the 
experiment  helpful  in  improving  your  own  decision? 

123456789 
Not  at  all  Very 

Helpful  Helpful 


A3. 2  Additional  question  asked  of  subjects  in  cognitive 
feedback  condition  only 

6.  To  what  extent  was  the  graphical  information  provided  on 
the  progress  of  the  project  helpful  in  improving  your  own 
decision? 

123456789 
Not  at  all  Very 

Helpful  Helpful 

***  PLEASE  MOVE  ON  TO  PROJECT  2  *** 
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A4  Information  provided  to  subject  for  second  project  -  order 
of  projects  varied  depending  on  subject's  group  assignment 


Project  2 


Management's  Initial  Project  Estimates 

Initial  Estimate  of  Project  Size:  397  Tasks 

Initial  Estimate  of  Project  Cost;  1,460  Man  Days 

Initial  Estimate  of  Project  Duration:  380  Days 

Maximum  Tolerable  Project  Duration:  441  Days 

A  project  is  considered  complete  when  there  are  less  than  40 
days  left  for  the  project  to  be  completed.  That  is,  (New 
Estimate  of  Project  Duration  -  Elapsed  Time)  <  40  days. 


Staffing  Level  Sought  (in  FTE) 

Please  enter  your  staffing  decisions,  i.e.,(In  Full  Time 
Equivalent)  below: 

Initial  Estimate:  _ 


Time 

elapsed 

- 

40  days:_ 

Time 

elapsed 

- 

80  days;_ 

Time 

elapsed 

- 

120 

days: 

Time 

elapsed 

- 

160 

days; 

Time 

elapsed 

- 

200 

days : 

Time 

elapsed 

- 

240 

days : 

Time 

elapsed 

- 

280 

days : 

Time 

elapsed 

- 

320 

days : 

Time 

elapsed 

- 

360 

days : 

Time 

elapsed 

- 

400 

days : 

Time 

elapsed 

- 

440 

days: 

Time 

elapsed 

- 

480 

days: 

Time 

elapsed 

- 

520 

days : 

Time  elapsed  -  560  days: _ 

Time  elapsed  -  600  days: _ 

Time  elapsed  -  640  days: _ 

***  WHEN  YOU  ARE  DONE.  PLEASE  CALL  FOR  A  LAB  ATTENDANT  *** 

A5  Questions  answered  by  subject  after  completion  of  second 
project  were  the  same  as  those  answered  after  the  first 
project 

***  PLEASE  MOVE  ON  TO  PROJECT  3  *** 
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h6  Information  provided  to  subject  for  third  project  -  order 
of  projects  varied  depending  on  subject' s  group  assignment 


Project  3 


Management's  Initial  Project  Estimates 


Initial  Estimate  of  Project  Size: 
Initial  Estimate  of  Project  Cost: 
Initial  Estimate  of  Project  Duration: 
Maximum  Tolerable  Project  Duration: 


1,866  Tasks 
2,972  Man  Days 
362  Days 
420  Days 


A  project  is  considered  complete  when  there  are  less  than  40 
days  left  for  the  project  to  be  completed.  That  is,  (New 
Estimate  of  Project  Duration  -  Elapsed  Time)  <  40  days. 


Staffing  Level  Sought  (in  FTE) 

Please  enter  your  staffing  decisions,  i.e.,(In  Full  Time 
Equivalent)  below: 

Initial  Estimate:  _ 


Time 

elapsed 

- 

40  days:_ 

Time 

elapsed 

- 

80  days;_ 

Time 

elapsed 

- 

120 

days:. 

Time 

elapsed 

- 

160 

days: 

Time 

elapsed 

- 

200 

days : 

Time 

elapsed 

- 

240 

days: 

Time 

elapsed 

- 

280 

days: 

Time 

elapsed 

- 

320 

days : 

Time 

elapsed 

- 

360 

days  : 

Time 

elapsed 

- 

400 

days : 

Time 

elapsed 

- 

440 

days: 

Time 

elapsed 

- 

480 

days ; 

Time 

elapsed 

- 

520 

days: 

Time 

elapsed 

- 

560 

days : 
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Time  elapsed  -  600  days: _ 

Time  elapsed  -  640  days: _ 

***  WHEN  YOU  ARE  DONE,  PLEASE  CALL  FOR  A  LAB  ATTENDANT  *** 


A1  Questions  answered  by  subject  after  complet ion  of  third 
project  were  the  same  as  those  answered  after  the  first 
project 


A8  Questions  answered  by  cognitive  feedback  subjects  after 
completion  of  entire  experiment  (subjects  in  the 
feedforward  and  outcome  feedback  answered  only  questions 
1,  7,  8,  9,  10,  11,  12,  13  and  14. 


1.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  project  status  report  (Y/N)? _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 


2.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  project  staffing  report  (Y/N)? _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 
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3.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  plot  on  project  size  (Y/N)? _ _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 


4.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  plot  on  staffing  and  schedule  (Y/N)? _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 


5.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  plot  on  workforce  mix  (Y/N)? _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 
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6.  In  the  projects  that  you  just  completed,  did  you 

(a)  Use  the  plot  on  workforce  productivity  (Y/N)? _ 

(b)  If  you  did,  please  describe  how  you  used  the 
information  to  make  the  staffing  decision. 


7.  Have  you  in  the  past,  participated  in  project  management 
(Y/N)?  _ 


8. 


If  YES,  to  what  extent  was  the  task  in  this  simulation 
similar  to  your  previous  experience? 


12  3 

Not  at  all 
Similar 


4  5  6  7  8  9 

Very 

Similar 


9. 


How  interesting  was 
12  3 

Not  at  all 
Interesting 


the  task  you  just  performed? 

4  5  6  7  8  9 

Very 

Interesting 


10.  How  serious  were  you  in  performing  the  task? 


12  3 

Not  at  all 
Serious 


4  5  6  7 


8  9 

Very 
Serious 


11.  How  clear  were  the  instructions  regarding  the  task, 
generally? 


1  2  3  4  5  6 

Not  at  all 
Clear 


7  8  9 

Very 
Clear 


12 .  How  easy  was  the  system  to  use? 

12345678 
Not  at  all 
Easy 
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9 

Very 

Easy 


13.  Please  give  us  some  information  about  yourself  (in 
absolute  confidence.  At  no  time  will  your  name  appear  in  the 
results.  The  data  will  only  be  used  in  an  aggregate 
statistical  sense). 

(a)  Curriculum  enrolled  in: _ 

(b)  Sex  _ 

( c )  Age  _ 

(d)  Full  time  work  experience 

(in  years)  _ 

(e)  How  long  ago  (in  years)  did 

you  complete  your 
undergraduate  education? _ 

(f)  How  familiar  are  you  with  computers,  generally? 

123456789 
Not  at  all  Very 

Familiar  Familiar 


(g)  How  many  hours  (per  week)  do  you  use  computers? 


14.  Your  general  comments  regarding  the  simulation: 


*** 


END  OF  SIMULATION 


•kitic 


Thank  you  for  your  participation. 
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